eXtended Cumulated Gain Metrics for the Evaluation of Content-oriented XML Retrieval

We propose and evaluate a family of metrics, named the eXtended Cumulated Gain (XCG) metrics, for the evaluation of content-oriented XML retrieval approaches. Our aim is to provide an evaluation framework that allows to consider the dependency that exists among XML document components and, in particular, incorporate mechanisms to reward the retrieval of so-called nearmisses and to address issues of overlap. Both system and user-oriented evaluation aspects are considered and both recall and precision-like qualities are measured. Another consideration is that the metrics should be flexible enough so that different models of user behaviour may be instantiated within. We evaluate the proposed XCG metrics, based on the INEX test collection, both with respect to their fidelity and reliability. For example, the effects of assessment variation and topic set size on evaluation stability are investigated, and upper and lower bounds of expected error rates are established. The evaluation demonstrates that the proposed XCG metrics are reliable, stable measures, and in particular that the novel metric, effort-precision and gain-recall (ep/gr), shows comparable behaviour to established measures like precision and recall.